install.packages("rpart")
Error in install.packages : Updating loaded packages
library(ROSE)
Loaded ROSE 0.0-3
TODO: Needs to chech accuracy, precision and f-measure
Nessa atividade você irá usar seus conhecimentos sobre classificação para prever quais candidatos à Câmara de Deputados serão eleitos nas eleições de 2014. De forma específica faremos o seguinte:
1 Há desbalanceamento das classes (isto é, uma classe tem muito mais instâncias que outra)? Em que proporção? Quais efeitos colaterais o desbalanceamento de classes pode causar no classificador? Como você poderia tratar isso? (10 pt.) 2 Treine: um modelo de KNN, regressão logística, uma árvore de decisão e um modelo de adaboost. Tune esses modelos usando validação cruzada e controle overfitting se necessário, considerando as particularidades de cada modelo. (20 pts.) 3 Reporte precision, recall e f-measure no treino e validação. Há uma grande diferença de desempenho no treino/validação? Como você avalia os resultados? Justifique sua resposta. (10 pt.) 4 Interprete as saídas dos modelos. Quais atributos parecem ser mais importantes de acordo com cada modelo? (20 pts.) 5 Envie seus melhores modelos à competição do Kaggle. Faça pelo menos uma submissão. Sugestões para melhorar o modelo: (20 pts.) >> 1 Experimente outros modelos (e.g. SVM, RandomForests e GradientBoosting). >> 2 Experimente balancear as classes, caso estejam desbalanceadas. >> 3 Experimente outras estratégias de ensembles (e.g. Stacking)
Os dados estão neste link: https://www.kaggle.com/c/ufcg-cdp-20182-lab3/data (Links para um site externo)Links para um site externo
Para a entrega envie o link no RPubs e os arquivos .Rmd com o código em R. Para as respostas esperamos explicações textuais e visualizações para cada questão.
Setting up workspace
setwd("~/git/data-analysis/lab03/")
Error in names(frame) <- `*vtmp*` : names() applied to a non-vector
Loading DATA
Our data frame will be the train.csv file, in which we’ll peform predictions models and test.csv will be used to Caggle challenge.
Here we gonna see the correlation between the variables, then will se the ones which has a strong correlation and remove, because keep both would be redundant for our prediction model.
data.correlation %>%
select(-partido,
-uf,-grau,-sexo) %>%
na.omit() %>%
ggcorr(palette = "RdBu",
color = "grey50",
label = TRUE, hjust = 1,
label_size = 3, size = 4,
nbreaks = 5, layout.exp = 7) +
ggtitle("Gráfico de correlação eleições 2006")
data in column(s) 'ocupacao', 'situacao' are not numeric and were ignored

We choosed to remove those three categoric variables in order to run the model, otherwise it would take too much time. But for a better result you could let them on the data. And also remove those variable which have strong correlation
test.kaggle <- test.kaggle %>%
select(-cargo, -nome, -ocupacao, total_despesa, -total_receita)
Error in is_character(x) : object 'cargo' not found
In the data would be better replace the NA for the column media, but we choosed replace by zero.
test.kaggle[is.na(test)] <- 0
Error in `[<-.data.frame`(`*tmp*`, is.na(test), value = 0) :
unsupported matrix index in replacement
As our target is to predict the variable situacao we need to see if our data is balanced, so what is the class distribution?
cleary unbalanced! So what should we do? We gonna balance it. There is some ways to balance data which are: >> 1. Undersampling That method reduces the number of observation from the majoritary class in order to balance the data set. >> 2. Oversampling This method increase the number of observation from the minoritary class and make it balanced. >> 3. Both Sampling Here it uses the technique 1 and 2 to make the data set balanced >> 4. ROSE Sampling Data synthetic generation and it provades a better stimation of original data.
Before balance it we gonna do a experiment. Let’s create a model and see how is goes whitout balance in order to predict and see accuracy to compare in the future
For tu build our models we gonna need data to train and test so we’ll divid the original data into train and test, 70% to raing and 30% to test.
Decision Tree whit unbalanced data
accuracy.meas(unbalanced.test$situacao, pred.treeimb[,2])
Call:
accuracy.meas(response = unbalanced.test$situacao, predicted = pred.treeimb[,
2])
Examples are labelled as positive when predicted is greater than 0.5
precision: 0.955
recall: 0.949
F: 0.476
roc.curve(unbalanced.test$situacao, pred.treeimb[,2], plotit = F)
Area under the curve (AUC): 0.890
Surprisely we’ve got a good precision and recall. Anyways let’s see how it goes whit balabced data.
Lets balance it, all data by using the 4 method ROSE Sampling which it gonna generate syntetich data.
table(data.rose$situacao)
nao_eleito eleito
3842 3780
YEAH!
It looks pretty balanced now. That is great, so we now gonna peform some models and avaliate its metrics.
Yes, we need to particionate our balanced data now, using the same schema before.
2 Treine: um modelo de KNN, regressão logística, uma árvore de decisão e um modelo de adaboost. Tune esses modelos usando validação cruzada e controle overfitting se necessário, considerando as particularidades de cada modelo. (20 pts.)
knn
First model is Knn.
k-nearest neighbour classification for test set from training set. For each row of the test set, the k nearest (in Euclidean distance) training set vectors are found, and the classification is decided by majority vote, with ties broken at random. If there are ties for the kth nearest vector, all candidates are included in the vote.
model.knn
k-Nearest Neighbors
5336 samples
17 predictors
2 classes: 'nao_eleito', 'eleito'
Pre-processing: centered (30), scaled (30), remove (49)
Resampling: Cross-Validated (10 fold, repeated 10 times)
Summary of sample sizes: 4802, 4803, 4803, 4802, 4802, 4802, ...
Resampling results across tuning parameters:
k Accuracy Kappa
5 0.8001493 0.5992494
7 0.7857002 0.5702039
9 0.7824398 0.5636275
Accuracy was used to select the optimal model using the largest value.
The final value used for the model was k = 5.
knn_cv
Accuracy Kappa
0.7979 0.5947
Logistic Regression
Second model to be build. That model aims to fit a regression curve, y= f(x), when y is a categorical variable.
model.logistic_reg
Boosted Logistic Regression
5336 samples
17 predictors
2 classes: 'nao_eleito', 'eleito'
Pre-processing: centered (30), scaled (30), remove (49)
Resampling: Cross-Validated (10 fold, repeated 10 times)
Summary of sample sizes: 4803, 4802, 4803, 4803, 4802, 4802, ...
Resampling results across tuning parameters:
nIter Accuracy Kappa
11 0.9311837 0.8623562
21 0.9626129 0.9252215
31 0.9642806 0.9285550
Accuracy was used to select the optimal model using the largest value.
The final value used for the model was nIter = 31.
logistic_reg_cv
Accuracy Kappa
0.9637 0.9274
Decision Tree
Third model Decision tree is a graph to represent choices and their results in form of a tree. The nodes in the graph represent an event or choice and the edges of the graph represent the decision rules or conditions.
new_index <- createDataPartition(data.rose$situacao, p = 0.7, list = FALSE)
new_train_data <- data.rose[index, ]
new_test_data <- data.rose[-index, ]
new_treeimb <- rpart(situacao ~ ., data = new_train_data)
new_pred.treeimb <- predict(new_treeimb, newdata = new_test_data)
accuracy.meas(new_test_data$situacao, new_pred.treeimb[,2])
model.tree_dec
CART
5336 samples
17 predictors
2 classes: 'nao_eleito', 'eleito'
No pre-processing
Resampling: Cross-Validated (10 fold, repeated 10 times)
Summary of sample sizes: 4802, 4802, 4802, 4803, 4802, 4803, ...
Resampling results across tuning parameters:
cp Accuracy Kappa
0.05442177 0.8897316 0.7794148
0.15873016 0.8324575 0.6642744
0.59372638 0.6332076 0.2614189
Accuracy was used to select the optimal model using the largest value.
The final value used for the model was cp = 0.05442177.
tree_cv
Accuracy Kappa
0.8648 0.7295
AdaBoost
Boosting is an ensemble technique that attempts to create a strong classifier from a number of weak classifiers.
model.adaboost <- train(situacao ~ media_receita + media_despesa,
data = data.rose,
trControl = fitControl,
method = 'adaboost',
metric = "Accuracy",
preProcess = preProcess)
model.adaboost
adaboost_prediction <- predict(model.logistic_reg,train)
adaboost_data <- data.frame(pred = logistic_reg_prediction, obs = train$situacao)
adaboost_cv <- round(defaultSummary(logistic_reg_data),digits = 4)
adaboost_cv
Which atribuite are most important to each model
4 Interprete as saídas dos modelos. Quais atributos parecem ser mais importantes de acordo com cada modelo? (20 pts.)
KNN
varImp(model.knn)
ROC curve variable importance
Importance
recursos_de_pessoas_juridicas 100.000
recursos_de_pessoas_fisicas 91.044
media_receita 81.484
quantidade_fornecedores 74.186
quantidade_despesas 73.979
quantidade_doadores 47.197
quantidade_doacoes 47.047
media_despesa 46.762
recursos_de_partido_politico 44.599
recursos_de_outros_candidatos.comites 40.221
recursos_proprios 36.430
grau 28.731
uf 26.858
partido 23.780
estado_civil 22.325
sexo 5.491
ano 0.000
Logistic Regression
varImp(model.logistic_reg)
ROC curve variable importance
Importance
recursos_de_pessoas_juridicas 100.000
recursos_de_pessoas_fisicas 91.044
media_receita 81.484
quantidade_fornecedores 74.186
quantidade_despesas 73.979
quantidade_doadores 47.197
quantidade_doacoes 47.047
media_despesa 46.762
recursos_de_partido_politico 44.599
recursos_de_outros_candidatos.comites 40.221
recursos_proprios 36.430
grau 28.731
uf 26.858
partido 23.780
estado_civil 22.325
sexo 5.491
ano 0.000
Decision Tree
varImp(model.tree_dec)
rpart variable importance
only 20 most important variables shown (out of 79)
Overall
recursos_de_pessoas_fisicas 100.00
quantidade_doacoes 89.12
recursos_de_pessoas_juridicas 78.10
quantidade_fornecedores 61.01
quantidade_despesas 58.30
quantidade_doadores 32.17
recursos_de_partido_politico 31.33
recursos_de_outros_candidatos.comites 30.86
ufAP 0.00
ufMT 0.00
ano 0.00
ufSP 0.00
partidoPCO 0.00
ufGO 0.00
ufAL 0.00
ufCE 0.00
`partidoPT do B` 0.00
ufRR 0.00
`estado_civilDIVORCIADO(A)` 0.00
partidoPR 0.00
AdaBoost
varImp(model.adaboost)
Kaggle challenge
As far we can see for ano and sexo we’ve a low outcome for importance, so those variables should be removed.
As propose in the activite we are going to use our best model to submite the votos prediction to the challenge in Kaggle.
5 Envie seus melhores modelos à competição do Kaggle. Faça pelo menos uma submissão. Sugestões para melhorar o modelo: (20 pts.) >> 1 Experimente outros modelos (e.g. SVM, RandomForests e GradientBoosting). >> 2 Experimente balancear as classes, caso estejam desbalanceadas. >> 3 Experimente outras estratégias de ensembles (e.g. Stacking)
LS0tCnRpdGxlOiAiUiBOb3RlYm9vayIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKYGBge3J9Cmluc3RhbGwucGFja2FnZXMoIm1sYmVuY2giKQppbnN0YWxsLnBhY2thZ2VzKCJDNTAiKQppbnN0YWxsLnBhY2thZ2VzKCJtYWdyaXR0ciIpCmluc3RhbGwucGFja2FnZXMoIlJPU0UiKQppbnN0YWxsLnBhY2thZ2VzKCJycGFydCIpCmBgYAoKYGBge3J9CmxpYnJhcnkoY2FyZXQpCmxpYnJhcnkobWxiZW5jaCkKbGlicmFyeShDNTApCmxpYnJhcnkoZHBseXIpCmxpYnJhcnkocGxvdGx5KQpsaWJyYXJ5KGNhcmV0KQpsaWJyYXJ5KFJPU0UpCmxpYnJhcnkocnBhcnQpCmxpYnJhcnkoR0dhbGx5KQpgYGAKI1RPRE86IE5lZWRzIHRvIGNoZWNoIGFjY3VyYWN5LCBwcmVjaXNpb24gYW5kIGYtbWVhc3VyZQoKTmVzc2EgYXRpdmlkYWRlIHZvY8OqIGlyw6EgdXNhciBzZXVzIGNvbmhlY2ltZW50b3Mgc29icmUgY2xhc3NpZmljYcOnw6NvIHBhcmEgcHJldmVyIHF1YWlzIGNhbmRpZGF0b3Mgw6AgQ8OibWFyYSBkZSBEZXB1dGFkb3Mgc2Vyw6NvIGVsZWl0b3MgbmFzIGVsZWnDp8O1ZXMgZGUgMjAxNC4gRGUgZm9ybWEgZXNwZWPDrWZpY2EgZmFyZW1vcyBvIHNlZ3VpbnRlOgoKPj4gMSBIw6EgZGVzYmFsYW5jZWFtZW50byBkYXMgY2xhc3NlcyAoaXN0byDDqSwgdW1hIGNsYXNzZSB0ZW0gbXVpdG8gbWFpcyBpbnN0w6JuY2lhcyBxdWUgb3V0cmEpPyBFbSBxdWUgcHJvcG9yw6fDo28/IFF1YWlzIGVmZWl0b3MgY29sYXRlcmFpcyBvIGRlc2JhbGFuY2VhbWVudG8gZGUgY2xhc3NlcyBwb2RlIGNhdXNhciBubyBjbGFzc2lmaWNhZG9yPyBDb21vIHZvY8OqIHBvZGVyaWEgdHJhdGFyIGlzc28/ICgxMCBwdC4pCj4+IDIgVHJlaW5lOiB1bSBtb2RlbG8gZGUgS05OLCByZWdyZXNzw6NvIGxvZ8Otc3RpY2EsIHVtYSDDoXJ2b3JlIGRlIGRlY2lzw6NvIGUgdW0gbW9kZWxvIGRlIGFkYWJvb3N0LiBUdW5lIGVzc2VzIG1vZGVsb3MgdXNhbmRvIHZhbGlkYcOnw6NvIGNydXphZGEgZSBjb250cm9sZSBvdmVyZml0dGluZyBzZSBuZWNlc3PDoXJpbywgY29uc2lkZXJhbmRvIGFzIHBhcnRpY3VsYXJpZGFkZXMgZGUgY2FkYSBtb2RlbG8uICAoMjAgcHRzLikKPj4gMyBSZXBvcnRlIHByZWNpc2lvbiwgcmVjYWxsIGUgZi1tZWFzdXJlIG5vIHRyZWlubyBlIHZhbGlkYcOnw6NvLiBIw6EgdW1hIGdyYW5kZSBkaWZlcmVuw6dhIGRlIGRlc2VtcGVuaG8gbm8gdHJlaW5vL3ZhbGlkYcOnw6NvPyBDb21vIHZvY8OqIGF2YWxpYSBvcyByZXN1bHRhZG9zPyBKdXN0aWZpcXVlIHN1YSByZXNwb3N0YS4gKDEwIHB0LikKPj4gNCBJbnRlcnByZXRlIGFzIHNhw61kYXMgZG9zIG1vZGVsb3MuIFF1YWlzIGF0cmlidXRvcyBwYXJlY2VtIHNlciBtYWlzIGltcG9ydGFudGVzIGRlIGFjb3JkbyBjb20gY2FkYSBtb2RlbG8/ICgyMCBwdHMuKQo+PiA1IEVudmllIHNldXMgbWVsaG9yZXMgbW9kZWxvcyDDoCBjb21wZXRpw6fDo28gZG8gS2FnZ2xlLiBGYcOnYSBwZWxvIG1lbm9zIHVtYSBzdWJtaXNzw6NvLiBTdWdlc3TDtWVzIHBhcmEgbWVsaG9yYXIgbyBtb2RlbG86ICgyMCBwdHMuKQo+Pj4+IDEgRXhwZXJpbWVudGUgb3V0cm9zIG1vZGVsb3MgKGUuZy4gU1ZNLCBSYW5kb21Gb3Jlc3RzIGUgR3JhZGllbnRCb29zdGluZykuCj4+Pj4gMiBFeHBlcmltZW50ZSBiYWxhbmNlYXIgYXMgY2xhc3NlcywgIGNhc28gZXN0ZWphbSBkZXNiYWxhbmNlYWRhcy4KPj4+PiAzIEV4cGVyaW1lbnRlIG91dHJhcyBlc3RyYXTDqWdpYXMgZGUgZW5zZW1ibGVzIChlLmcuIFN0YWNraW5nKQoKT3MgZGFkb3MgZXN0w6NvIG5lc3RlIGxpbms6IGh0dHBzOi8vd3d3LmthZ2dsZS5jb20vYy91ZmNnLWNkcC0yMDE4Mi1sYWIzL2RhdGEgKExpbmtzIHBhcmEgdW0gc2l0ZSBleHRlcm5vKUxpbmtzIHBhcmEgdW0gc2l0ZSBleHRlcm5vCgpQYXJhIGEgZW50cmVnYSBlbnZpZSBvIGxpbmsgbm8gUlB1YnMgZSBvcyBhcnF1aXZvcyAuUm1kIGNvbSBvIGPDs2RpZ28gZW0gUi4gUGFyYSBhcyByZXNwb3N0YXMgZXNwZXJhbW9zIGV4cGxpY2HDp8O1ZXMgdGV4dHVhaXMgZSB2aXN1YWxpemHDp8O1ZXMgcGFyYSBjYWRhIHF1ZXN0w6NvLgoKU2V0dGluZyB1cCB3b3Jrc3BhY2UKYGBge3J9CnNldHdkKCJ+L2dpdC9kYXRhLWFuYWx5c2lzL2xhYjAzLyIpCmBgYAoKCkxvYWRpbmcgREFUQQoKT3VyIGRhdGEgZnJhbWUgd2lsbCBiZSB0aGUgdHJhaW4uY3N2IGZpbGUsIGluIHdoaWNoIHdlJ2xsIHBlZm9ybSBwcmVkaWN0aW9ucyBtb2RlbHMKYW5kIHRlc3QuY3N2IHdpbGwgYmUgdXNlZCB0byBDYWdnbGUgY2hhbGxlbmdlLgoKYGBge3J9CmRhdGEgPC0gcmVhZC5jc3YoImRhdGEvYWxsL3RyYWluLmNzdiIpCnRlc3Qua2FnZ2xlIDwtIHJlYWQuY3N2KCJkYXRhL2FsbC90ZXN0LmNzdiIpCmBgYAoKSGVyZSB3ZSBnb25uYSBzZWUgdGhlIGNvcnJlbGF0aW9uIGJldHdlZW4gdGhlIHZhcmlhYmxlcywgdGhlbiB3aWxsIHNlIHRoZSBvbmVzIHdoaWNoIGhhcyBhIHN0cm9uZyBjb3JyZWxhdGlvbiBhbmQgcmVtb3ZlLCBiZWNhdXNlIGtlZXAgYm90aCB3b3VsZCBiZSByZWR1bmRhbnQgZm9yIG91ciBwcmVkaWN0aW9uIG1vZGVsLgoKYGBge3J9CmRhdGEuY29ycmVsYXRpb24xIDwtIGRhdGEgJT4lIHNlbGVjdCgtYyhzZXF1ZW5jaWFsX2NhbmRpZGF0bywgbm9tZSwgZXN0YWRvX2NpdmlsLCBhbm8sIGNhcmdvKSkKCmRhdGEuY29ycmVsYXRpb24gPC0gZGF0YS5jb3JyZWxhdGlvbjEgICU+JQogIG11dGF0ZShzaXR1YWNhbyA9IGFzLmZhY3RvcihzaXR1YWNhbykpICU+JQogIG11dGF0ZSh1ZiA9IGFzLmZhY3Rvcih1ZikpICU+JQogIG11dGF0ZShwYXJ0aWRvID0gYXMuZmFjdG9yKHBhcnRpZG8pKSAlPiUKICBtdXRhdGUoc2V4byA9IGFzLmZhY3RvcihzZXhvKSkgJT4lCiAgbXV0YXRlKGdyYXUgPSBhcy5mYWN0b3IoZ3JhdSkpICU+JQogIG11dGF0ZShvY3VwYWNhbyA9IGFzLmZhY3RvcihvY3VwYWNhbykpCgpkYXRhLmNvcnJlbGF0aW9uICU+JSAKICBzZWxlY3QoLXBhcnRpZG8sCiAgICAgICAgIC11ZiwtZ3JhdSwtc2V4bykgJT4lCiAgbmEub21pdCgpICU+JQogIGdnY29ycihwYWxldHRlID0gIlJkQnUiLAogICAgICAgICBjb2xvciA9ICJncmV5NTAiLAogICAgICAgICBsYWJlbCA9IFRSVUUsIGhqdXN0ID0gMSwKICAgICAgICAgbGFiZWxfc2l6ZSA9IDMsIHNpemUgPSA0LAogICAgICAgICBuYnJlYWtzID0gNSwgbGF5b3V0LmV4cCA9IDcpICsKICBnZ3RpdGxlKCJHcsOhZmljbyBkZSBjb3JyZWxhw6fDo28gZWxlacOnw7VlcyAyMDA2IikKYGBgCgoKV2UgY2hvb3NlZCB0byByZW1vdmUgdGhvc2UgdGhyZWUgY2F0ZWdvcmljIHZhcmlhYmxlcyBpbiBvcmRlciB0byBydW4gdGhlIG1vZGVsLCBvdGhlcndpc2UgaXQgd291bGQgdGFrZSB0b28gbXVjaCB0aW1lLiBCdXQgZm9yIGEgYmV0dGVyIHJlc3VsdCB5b3UgY291bGQgbGV0IHRoZW0gb24gdGhlIGRhdGEuIEFuZCBhbHNvIHJlbW92ZSB0aG9zZSB2YXJpYWJsZSB3aGljaCBoYXZlIHN0cm9uZyBjb3JyZWxhdGlvbgoKYGBge3J9CmRhdGEgPC0gZGF0YSAlPiUKICBzZWxlY3QoLWNhcmdvLCAtbm9tZSwgLW9jdXBhY2FvLCAtc2V4bywgLXRvdGFsX2Rlc3Blc2EsIC10b3RhbF9yZWNlaXRhLCAtc2VxdWVuY2lhbF9jYW5kaWRhdG8gKQp0ZXN0LmthZ2dsZSA8LSB0ZXN0LmthZ2dsZSAlPiUKICBzZWxlY3QoLWNhcmdvLCAtbm9tZSwgLW9jdXBhY2FvLCB0b3RhbF9kZXNwZXNhLCAtdG90YWxfcmVjZWl0YSkKYGBgCgpJbiB0aGUgZGF0YSB3b3VsZCBiZSBiZXR0ZXIgcmVwbGFjZSB0aGUgTkEgZm9yIHRoZSBjb2x1bW4gbWVkaWEsIGJ1dCB3ZSBjaG9vc2VkIHJlcGxhY2UgYnkgemVyby4KCgpgYGB7cn0KZGF0YVtpcy5uYShkYXRhKV0gPC0gMAp0ZXN0LmthZ2dsZVtpcy5uYSh0ZXN0LmthZ2dsZSldIDwtIDAKYGBgCgpBcyBvdXIgdGFyZ2V0IGlzIHRvIHByZWRpY3QgdGhlIHZhcmlhYmxlICpzaXR1YWNhbyogd2UgbmVlZCB0byBzZWUgaWYgb3VyIGRhdGEgaXMgYmFsYW5jZWQsIHNvIHdoYXQgaXMgdGhlIGNsYXNzIGRpc3RyaWJ1dGlvbj8gCgoKYGBge3J9CmRhdGFfY2xhc3NfZGVzdHJpYnV0aW9uIDwtIGRhdGEgJT4lIGdyb3VwX2J5KHNpdHVhY2FvKSAlPiUgc3VtbWFyaXplKGNsYXNzX2NvdW50ID0gbigpKQpwIDwtIHBsb3RfbHkoZGF0YV9jbGFzc19kZXN0cmlidXRpb24sIHggPSB+c2l0dWFjYW8sIHkgPSB+Y2xhc3NfY291bnQsIHR5cGUgPSAnYmFyJywKICAgICAgICBtYXJrZXIgPSBsaXN0KGNvbG9yID0gYygncmdiYSgyMDQsMjA0LDIwNCwxKScsICdyZ2JhKDIyMiw0NSwzOCwwLjgpJykpKSAlPiUKICBsYXlvdXQodGl0bGUgPSAiQ2xhc3MgQmFsYW5jZSIsCiAgICAgICAgIHhheGlzID0gbGlzdCh0aXRsZSA9ICJTaXR1YXRpb24iKSwKICAgICAgICAgeWF4aXMgPSBsaXN0KHRpdGxlID0gIkNvdW50IikpCnAKYGBgCgpjbGVhcnkgdW5iYWxhbmNlZCEKU28gd2hhdCBzaG91bGQgd2UgZG8/IFdlIGdvbm5hIGJhbGFuY2UgaXQuClRoZXJlIGlzIHNvbWUgd2F5cyB0byBiYWxhbmNlIGRhdGEgd2hpY2ggYXJlOgo+PiAxLiBVbmRlcnNhbXBsaW5nClRoYXQgbWV0aG9kIHJlZHVjZXMgdGhlIG51bWJlciBvZiBvYnNlcnZhdGlvbiBmcm9tIHRoZSBtYWpvcml0YXJ5IGNsYXNzIGluIG9yZGVyIHRvIGJhbGFuY2UgdGhlIGRhdGEgc2V0Lgo+PiAyLiBPdmVyc2FtcGxpbmcKVGhpcyBtZXRob2QgaW5jcmVhc2UgdGhlIG51bWJlciBvZiBvYnNlcnZhdGlvbiBmcm9tIHRoZSBtaW5vcml0YXJ5IGNsYXNzIGFuZCBtYWtlIGl0IGJhbGFuY2VkLgo+PiAzLiBCb3RoIFNhbXBsaW5nCkhlcmUgaXQgdXNlcyB0aGUgdGVjaG5pcXVlIDEgYW5kIDIgdG8gbWFrZSB0aGUgZGF0YSBzZXQgYmFsYW5jZWQgCj4+IDQuIFJPU0UgU2FtcGxpbmcKRGF0YSBzeW50aGV0aWMgZ2VuZXJhdGlvbiBhbmQgaXQgcHJvdmFkZXMgYSBiZXR0ZXIgc3RpbWF0aW9uIG9mIG9yaWdpbmFsIGRhdGEuCgoKCgpCZWZvcmUgYmFsYW5jZSBpdCB3ZSBnb25uYSBkbyBhIGV4cGVyaW1lbnQuIExldCdzIGNyZWF0ZSBhIG1vZGVsIGFuZCBzZWUgaG93IGlzIGdvZXMgd2hpdG91dCBiYWxhbmNlIGluIG9yZGVyIHRvIHByZWRpY3QgYW5kIHNlZSBhY2N1cmFjeSB0byBjb21wYXJlIGluIHRoZSBmdXR1cmUKCgoKRm9yIHR1IGJ1aWxkIG91ciBtb2RlbHMgd2UgZ29ubmEgbmVlZCBkYXRhIHRvIHRyYWluIGFuZCB0ZXN0IHNvIHdlJ2xsIGRpdmlkIHRoZSBvcmlnaW5hbCBkYXRhIGludG8gdHJhaW4gYW5kIHRlc3QsIDcwJSB0byByYWluZyBhbmQgMzAlIHRvIHRlc3QuCgpgYGB7cn0Kc2V0LnNlZWQoNDIpCmluZGV4IDwtIGNyZWF0ZURhdGFQYXJ0aXRpb24oZGF0YSRzaXR1YWNhbywgcCA9IDAuNywgbGlzdCA9IEZBTFNFKQp1bmJhbGFuY2VkLnRyYWluIDwtIGRhdGFbaW5kZXgsIF0KdW5iYWxhbmNlZC50ZXN0IDwtIGRhdGFbLWluZGV4LCBdCmBgYAoKCkRlY2lzaW9uIFRyZWUgd2hpdCB1bmJhbGFuY2VkIGRhdGEKCmBgYHtyfQoKdHJlZWltYiA8LSBycGFydChzaXR1YWNhbyB+IC4sIGRhdGEgPSB1bmJhbGFuY2VkLnRyYWluKQpwcmVkLnRyZWVpbWIgPC0gcHJlZGljdCh0cmVlaW1iLCBuZXdkYXRhID0gdW5iYWxhbmNlZC50ZXN0KQoKYWNjdXJhY3kubWVhcyh1bmJhbGFuY2VkLnRlc3Qkc2l0dWFjYW8sIHByZWQudHJlZWltYlssMl0pCmBgYApgYGB7cn0Kcm9jLmN1cnZlKHVuYmFsYW5jZWQudGVzdCRzaXR1YWNhbywgcHJlZC50cmVlaW1iWywyXSwgcGxvdGl0ID0gRikKYGBgCgpTdXJwcmlzZWx5IHdlJ3ZlIGdvdCBhIGdvb2QgcHJlY2lzaW9uIGFuZCByZWNhbGwuIEFueXdheXMgbGV0J3Mgc2VlIGhvdyBpdCBnb2VzIHdoaXQgYmFsYWJjZWQgZGF0YS4KCgoKTGV0cyBiYWxhbmNlIGl0LCBhbGwgZGF0YSBieSB1c2luZyB0aGUgNCBtZXRob2QgUk9TRSBTYW1wbGluZyB3aGljaCBpdCBnb25uYSBnZW5lcmF0ZSBzeW50ZXRpY2ggZGF0YS4KCgpgYGB7cn0KZGF0YS5yb3NlIDwtIFJPU0Uoc2l0dWFjYW8gfiAuLCBkYXRhID0gZGF0YSwgc2VlZCA9IDEpJGRhdGEKdGFibGUoZGF0YS5yb3NlJHNpdHVhY2FvKQpgYGAKWUVBSCEKCkl0IGxvb2tzIHByZXR0eSBiYWxhbmNlZCBub3cuIFRoYXQgaXMgZ3JlYXQsIHNvIHdlIG5vdyBnb25uYSBwZWZvcm0gc29tZSBtb2RlbHMgYW5kIGF2YWxpYXRlIGl0cyBtZXRyaWNzLgoKWWVzLCB3ZSBuZWVkIHRvIHBhcnRpY2lvbmF0ZSBvdXIgYmFsYW5jZWQgZGF0YSBub3csIHVzaW5nIHRoZSBzYW1lIHNjaGVtYSBiZWZvcmUuCgpgYGB7cn0Kc2V0LnNlZWQoNDIpCmluZGV4IDwtIGNyZWF0ZURhdGFQYXJ0aXRpb24oZGF0YS5yb3NlJHNpdHVhY2FvLCBwID0gMC43LCBsaXN0ID0gRkFMU0UpCnRyYWluIDwtIGRhdGEucm9zZVtpbmRleCwgXQp0ZXN0IDwtIGRhdGEucm9zZVstaW5kZXgsIF0KYGBgCgo+PiAyIFRyZWluZTogdW0gbW9kZWxvIGRlIEtOTiwgcmVncmVzc8OjbyBsb2fDrXN0aWNhLCB1bWEgw6Fydm9yZSBkZSBkZWNpc8OjbyBlIHVtIG1vZGVsbyBkZSBhZGFib29zdC4gVHVuZSBlc3NlcyBtb2RlbG9zIHVzYW5kbyB2YWxpZGHDp8OjbyBjcnV6YWRhIGUgY29udHJvbGUgb3ZlcmZpdHRpbmcgc2UgbmVjZXNzw6FyaW8sIGNvbnNpZGVyYW5kbyBhcyBwYXJ0aWN1bGFyaWRhZGVzIGRlIGNhZGEgbW9kZWxvLiAgKDIwIHB0cy4pCgoja25uCkZpcnN0IG1vZGVsIGlzIEtubi4KCmstbmVhcmVzdCBuZWlnaGJvdXIgY2xhc3NpZmljYXRpb24gZm9yIHRlc3Qgc2V0IGZyb20gdHJhaW5pbmcgc2V0LiBGb3IgZWFjaCByb3cgb2YgdGhlIHRlc3Qgc2V0LCB0aGUgayBuZWFyZXN0IChpbiBFdWNsaWRlYW4gZGlzdGFuY2UpIHRyYWluaW5nIHNldCB2ZWN0b3JzIGFyZSBmb3VuZCwgYW5kIHRoZSBjbGFzc2lmaWNhdGlvbiBpcyBkZWNpZGVkIGJ5IG1ham9yaXR5IHZvdGUsIHdpdGggdGllcyBicm9rZW4gYXQgcmFuZG9tLiBJZiB0aGVyZSBhcmUgdGllcyBmb3IgdGhlIGt0aCBuZWFyZXN0IHZlY3RvciwgYWxsIGNhbmRpZGF0ZXMgYXJlIGluY2x1ZGVkIGluIHRoZSB2b3RlLgoKCmBgYHtyfQpmaXRDb250cm9sIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAicmVwZWF0ZWRjdiIsIAogICAgICAgICAgICAgICAgICAgICAgICAgICBudW1iZXIgPSAxMCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgcmVwZWF0cyA9IDEwKQoKcHJlUHJvY2VzcyA9IGMoImNlbnRlciIsICJzY2FsZSIsIm56diIgKQpgYGAKCmBgYHtyfQptb2RlbC5rbm4gPC0gdHJhaW4oc2l0dWFjYW8gfiAuLCAKICAgICAgICAgICAgICAgZGF0YSA9IHRyYWluLAogICAgICAgICAgICAgICB0ckNvbnRyb2wgPSBmaXRDb250cm9sLAogICAgICAgICAgICAgICBtZXRob2QgPSAia25uIiwgIyBwb2RlIHNlciAnbGFzc28nbGRmCiAgICAgICAgICAgICAgIG1ldHJpYyA9ICJBY2N1cmFjeSIsCiAgICAgICAgICAgICAgIHByZVByb2Nlc3MgPSBwcmVQcm9jZXNzKQoKbW9kZWwua25uCmBgYAoKCmBgYHtyfQprbm5fcHJlZGljdGlvbiA8LSBwcmVkaWN0KG1vZGVsLmtubix0ZXN0KQoKa25uX2RhdGEgPC0gZGF0YS5mcmFtZShwcmVkID0ga25uX3ByZWRpY3Rpb24sIG9icyA9IHRlc3Qkc2l0dWFjYW8pCgprbm5fY3YgPC0gcm91bmQoZGVmYXVsdFN1bW1hcnkoa25uX2RhdGEpLGRpZ2l0cyA9IDQpCgprbm5fY3YKYGBgCgojTG9naXN0aWMgUmVncmVzc2lvbgpTZWNvbmQgbW9kZWwgdG8gYmUgYnVpbGQuClRoYXQgbW9kZWwgYWltcyB0byBmaXQgYSByZWdyZXNzaW9uIGN1cnZlLCB5PSBmKHgpLCB3aGVuIHkgaXMgYSBjYXRlZ29yaWNhbCB2YXJpYWJsZS4KCgpgYGB7cn0KbW9kZWwubG9naXN0aWNfcmVnIDwtIHRyYWluKHNpdHVhY2FvIH4gLiwgCiAgICAgICAgICAgICAgIGRhdGEgPSB0cmFpbiwKICAgICAgICAgICAgICAgdHJDb250cm9sID0gZml0Q29udHJvbCwKICAgICAgICAgICAgICAgbWV0aG9kID0gJ0xvZ2l0Qm9vc3QnLCAKICAgICAgICAgICAgICAgbWV0cmljID0gIkFjY3VyYWN5IiwKICAgICAgICAgICAgICAgcHJlUHJvY2VzcyA9IHByZVByb2Nlc3MpCgptb2RlbC5sb2dpc3RpY19yZWcKYGBgCgoKCmBgYHtyfQpsb2dpc3RpY19yZWdfcHJlZGljdGlvbiA8LSBwcmVkaWN0KG1vZGVsLmxvZ2lzdGljX3JlZyx0ZXN0KQoKbG9naXN0aWNfcmVnX2RhdGEgPC0gZGF0YS5mcmFtZShwcmVkID0gbG9naXN0aWNfcmVnX3ByZWRpY3Rpb24sIG9icyA9IHRlc3Qkc2l0dWFjYW8pCgpsb2dpc3RpY19yZWdfY3YgPC0gcm91bmQoZGVmYXVsdFN1bW1hcnkobG9naXN0aWNfcmVnX2RhdGEpLGRpZ2l0cyA9IDQpCgpsb2dpc3RpY19yZWdfY3YKCmBgYAoKICAKI0RlY2lzaW9uIFRyZWUKVGhpcmQgbW9kZWwgCkRlY2lzaW9uIHRyZWUgaXMgYSBncmFwaCB0byByZXByZXNlbnQgY2hvaWNlcyBhbmQgdGhlaXIgcmVzdWx0cyBpbiBmb3JtIG9mIGEgdHJlZS4gVGhlIG5vZGVzIGluIHRoZSBncmFwaCByZXByZXNlbnQgYW4gZXZlbnQgb3IgY2hvaWNlIGFuZCB0aGUgZWRnZXMgb2YgdGhlIGdyYXBoIHJlcHJlc2VudCB0aGUgZGVjaXNpb24gcnVsZXMgb3IgY29uZGl0aW9ucy4KCmBgYHtyfQpuZXdfaW5kZXggPC0gY3JlYXRlRGF0YVBhcnRpdGlvbihkYXRhLnJvc2Ukc2l0dWFjYW8sIHAgPSAwLjcsIGxpc3QgPSBGQUxTRSkKbmV3X3RyYWluX2RhdGEgPC0gZGF0YS5yb3NlW2luZGV4LCBdCm5ld190ZXN0X2RhdGEgIDwtIGRhdGEucm9zZVstaW5kZXgsIF0KCgpuZXdfdHJlZWltYiA8LSBycGFydChzaXR1YWNhbyB+IC4sIGRhdGEgPSBuZXdfdHJhaW5fZGF0YSkKbmV3X3ByZWQudHJlZWltYiA8LSBwcmVkaWN0KG5ld190cmVlaW1iLCBuZXdkYXRhID0gbmV3X3Rlc3RfZGF0YSkKCgphY2N1cmFjeS5tZWFzKG5ld190ZXN0X2RhdGEkc2l0dWFjYW8sIG5ld19wcmVkLnRyZWVpbWJbLDJdKQpgYGAKCmBgYHtyfQptb2RlbC50cmVlX2RlYyA8LSB0cmFpbihzaXR1YWNhbyB+IC4sCiAgICAgICAgICAgICAgICBkYXRhPSB0cmFpbiwgCiAgICAgICAgICAgICAgICBtZXRob2QgPSAicnBhcnQiLAogICAgICAgICAgICAgICAgdHJDb250cm9sID0gZml0Q29udHJvbCwKICAgICAgICAgICAgICAgIGNwPTAuMDAxLCAgCiAgICAgICAgICAgICAgICBtZXRyaWMgPSAiQWNjdXJhY3kiLAogICAgICAgICAgICAgICAgbWF4ZGVwdGg9MjApCm1vZGVsLnRyZWVfZGVjCmBgYAoKYGBge3J9CnRyZWVfcHJlZGljdGlvbiA8LSBwcmVkaWN0KG1vZGVsLnRyZWVfZGVjLHRlc3QpCgp0cmVlX2RhdGEgPC0gZGF0YS5mcmFtZShwcmVkID0gdHJlZV9wcmVkaWN0aW9uLCBvYnMgPSB0ZXN0JHNpdHVhY2FvKQoKdHJlZV9jdiA8LSByb3VuZChkZWZhdWx0U3VtbWFyeSh0cmVlX2RhdGEpLGRpZ2l0cyA9IDQpCgp0cmVlX2N2CgpgYGAKCgojQWRhQm9vc3QKQm9vc3RpbmcgaXMgYW4gZW5zZW1ibGUgdGVjaG5pcXVlIHRoYXQgYXR0ZW1wdHMgdG8gY3JlYXRlIGEgc3Ryb25nIGNsYXNzaWZpZXIgZnJvbSBhIG51bWJlciBvZiB3ZWFrIGNsYXNzaWZpZXJzLgoKYGBge3J9CgoKbW9kZWwuYWRhYm9vc3QgPC0gdHJhaW4oc2l0dWFjYW8gfiBtZWRpYV9yZWNlaXRhICsgbWVkaWFfZGVzcGVzYSwgCiAgICAgICAgICAgICAgIGRhdGEgPSBkYXRhLnJvc2UsCiAgICAgICAgICAgICAgIHRyQ29udHJvbCA9IGZpdENvbnRyb2wsCiAgICAgICAgICAgICAgIG1ldGhvZCA9ICdhZGFib29zdCcsIAogICAgICAgICAgICAgICBtZXRyaWMgPSAiQWNjdXJhY3kiLAogICAgICAgICAgICAgICBwcmVQcm9jZXNzID0gcHJlUHJvY2VzcykKCm1vZGVsLmFkYWJvb3N0CgoKYGBgCgoKYGBge3J9CmFkYWJvb3N0X3ByZWRpY3Rpb24gPC0gcHJlZGljdChtb2RlbC5sb2dpc3RpY19yZWcsdHJhaW4pCgphZGFib29zdF9kYXRhIDwtIGRhdGEuZnJhbWUocHJlZCA9IGxvZ2lzdGljX3JlZ19wcmVkaWN0aW9uLCBvYnMgPSB0cmFpbiRzaXR1YWNhbykKCmFkYWJvb3N0X2N2IDwtIHJvdW5kKGRlZmF1bHRTdW1tYXJ5KGxvZ2lzdGljX3JlZ19kYXRhKSxkaWdpdHMgPSA0KQoKYWRhYm9vc3RfY3YKCmBgYAoKIyNXaGljaCBhdHJpYnVpdGUgYXJlIG1vc3QgaW1wb3J0YW50IHRvIGVhY2ggbW9kZWwKCj4+IDQgSW50ZXJwcmV0ZSBhcyBzYcOtZGFzIGRvcyBtb2RlbG9zLiBRdWFpcyBhdHJpYnV0b3MgcGFyZWNlbSBzZXIgbWFpcyBpbXBvcnRhbnRlcyBkZSBhY29yZG8gY29tIGNhZGEgbW9kZWxvPyAoMjAgcHRzLikKCiMjI0tOTgpgYGB7cn0KdmFySW1wKG1vZGVsLmtubikKYGBgCgoKIyMjTG9naXN0aWMgUmVncmVzc2lvbgoKYGBge3J9CnZhckltcChtb2RlbC5sb2dpc3RpY19yZWcpCmBgYAoKIyMjRGVjaXNpb24gVHJlZQoKYGBge3J9CnZhckltcChtb2RlbC50cmVlX2RlYykKYGBgCgojIyNBZGFCb29zdAoKYGBge3J9CnZhckltcChtb2RlbC5hZGFib29zdCkKYGBgCgoKIyMgS2FnZ2xlIGNoYWxsZW5nZQpBcyBmYXIgd2UgY2FuIHNlZSBmb3IgKmFubyogYW5kICpzZXhvKiB3ZSd2ZSBhIGxvdyBvdXRjb21lIGZvciBpbXBvcnRhbmNlLCBzbyB0aG9zZSB2YXJpYWJsZXMgc2hvdWxkIGJlIHJlbW92ZWQuCgoKCkFzIHByb3Bvc2UgaW4gdGhlIGFjdGl2aXRlIHdlIGFyZSBnb2luZyB0byB1c2Ugb3VyIGJlc3QgbW9kZWwgdG8gc3VibWl0ZSB0aGUgdm90b3MgcHJlZGljdGlvbiB0byB0aGUgY2hhbGxlbmdlIGluIEthZ2dsZS4gCgo+PiA1IEVudmllIHNldXMgbWVsaG9yZXMgbW9kZWxvcyDDoCBjb21wZXRpw6fDo28gZG8gS2FnZ2xlLiBGYcOnYSBwZWxvIG1lbm9zIHVtYSBzdWJtaXNzw6NvLiBTdWdlc3TDtWVzIHBhcmEgbWVsaG9yYXIgbyBtb2RlbG86ICgyMCBwdHMuKQo+Pj4+IDEgRXhwZXJpbWVudGUgb3V0cm9zIG1vZGVsb3MgKGUuZy4gU1ZNLCBSYW5kb21Gb3Jlc3RzIGUgR3JhZGllbnRCb29zdGluZykuCj4+Pj4gMiBFeHBlcmltZW50ZSBiYWxhbmNlYXIgYXMgY2xhc3NlcywgIGNhc28gZXN0ZWphbSBkZXNiYWxhbmNlYWRhcy4KPj4+PiAzIEV4cGVyaW1lbnRlIG91dHJhcyBlc3RyYXTDqWdpYXMgZGUgZW5zZW1ibGVzIChlLmcuIFN0YWNraW5nKQoKI1RPRE86IEl0IG5lZWRzIHRvIGJlIHJldmlzaWRlIAoKCgoKYGBge3IgLHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9Cm1vZGVsLmxvZ2lzdGljX3JlZyAkeGxldmVsc1tbIm9jdXBhY2FvIl1dIDwtIHVuaW9uKG1vZGVsLmxvZ2lzdGljX3JlZyR4bGV2ZWxzW1sib2N1cGFjYW8iXV0sIGxldmVscyh0ZXN0LmthZ2dsZSRvY3VwYWNhbykpCnByZWRpY3Rpb25fIDwtIHByZWRpY3QobW9kZWwubG9naXN0aWNfcmVnICwgdGVzdC5rYWdnbGUpCklEIDwtIHRlc3Qua2FnZ2xlICU+JQogIHNlbGVjdChzZXF1ZW5jaWFsX2NhbmRpZGF0bykKY29sbmFtZXMoSUQpW2NvbG5hbWVzKElEKT09InNlcXVlbmNpYWxfY2FuZGlkYXRvIl0gPC0gIklEIgpwcmVkaWN0ZWRfZmlsZSA8LSBJRApwcmVkaWN0ZWRfZmlsZSRzaXR1YWNhbyA8LSBwcmVkaWN0aW9uXwpwcmVkaWN0ZWRfZmlsZSRzaXR1YWNhb1twcmVkaWN0ZWRfZmlsZSRzaXR1YWNhbyA8IDBdIDwtIDAKd3JpdGUuY3N2KHByZWRpY3RlZF9maWxlLCAic2FtcGxlX3N1Ym1pc3Npb24uY3N2Iiwgcm93Lm5hbWVzPUZBTFNFKQpgYGAKCgoKCgoKCnVzZWZ1bGwgbGlua3M6Cmh0dHA6Ly93d3cudHJlc2VsbGUuY29tL2Jsb2cvaGFuZGxlLWNsYXNzLWltYmFsYW5jZS1kYXRhLXdpdGgtci8KaHR0cHM6Ly93d3cuYW5hbHl0aWNzdmlkaHlhLmNvbS9ibG9nLzIwMTYvMDMvcHJhY3RpY2FsLWd1aWRlLWRlYWwtaW1iYWxhbmNlZC1jbGFzc2lmaWNhdGlvbi1wcm9ibGVtcy8KaHR0cHM6Ly9zaGlyaW5nLmdpdGh1Yi5pby9tYWNoaW5lX2xlYXJuaW5nLzIwMTcvMDQvMDIvdW5iYWxhbmNlZCAKCgoK